Real-time dashboards, alerts and analytics for a log intelligence system

ABSTRACT

This disclosure describes how data supporting real-time reporting services can be cached during a log intake process. In particular, instead of caching all the log data being generated by an operational system, only the log data relevant to existing queries associated with the real-time reporting services are cached. In some embodiments, only particular metrics contained within the log data are stored for rapid access by the real-time reporting services.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/807,187 filed on Mar. 3, 2020 which is entitled “REAL-TIMEDASHBOARDS, ALERTS AND ANALYTICS FOR A LOG INTELLIGENCE SYSTEM” andwhich claims the benefit under 35 U.S.C. 119(a)-(d) to ForeignApplication Serial No. 201941051280 filed in India entitled “REAL-TIMEDASHBOARDS, ALERTS AND ANALYTICS FOR A LOG INTELLIGENCE SYSTEM” on Dec.11, 2019, by VMWARE, Inc., each of which are herein incorporated byreference in their entireties and for all purposes.

FIELD

The present disclosure relates generally to caching specific datasetsfor populating and allowing quick updates of one or more real-timereporting services.

BACKGROUND

Log intelligence systems serve the distinct purpose of providingactionable insights from time-stamped events generated by multiplesystems. Some capabilities of log intelligence systems include:ingestion of high volumes of data at high throughput; parsing androuting the data to a storage layer where it can be indexed; andproviding user-facing functionalities such as real-time queries, alerts,dashboards and analytics. These systems rely on querying indexed logsstored in a text indexing system to cater to most user-facingfunctionalities. Unfortunately, leveraging this method for triggeringalerts, and generating dashboard analytics can be quite inefficient.This method becomes even less efficient when the indexed logs aredistributed across different shards. For this reason, methods of drivingreal-time reporting services such as dashboards and alerts withnon-indexed data is desirable.

SUMMARY

This disclosure describes ways in which data supporting real-timedashboard services can be cached during a log intake process. Systemsand methods for filtering and storing data pertinent to queriessupporting the dashboard services during the log intake process aredescribed.

A computer implemented method for displaying metrics associated with logdata is described and includes: receiving a stream of log data beinggenerated by an operational system; forwarding the stream of log data toa first data plane and to a constraint plane; storing the stream of logdata in the first data plane; extracting a subset of the stream of logdata at the constraint plane in accordance with a set of rules based onpredefined queries of a real-time reporting service; saving the subsetof the stream of log data to a second data plane; transmitting one ormore metrics included in the subset of the stream of log data to thereal-time reporting service; and providing the one or more metrics fromthe subset of the stream of log data to a user of the real-timereporting service.

A non-transitory computer-readable storage medium is described. Thecomputer readable storage medium includes instructions configured to beexecuted by one or more processors of a computing device and to causethe computing device to carry out steps that include: receiving a streamof log data being generated by an operational system; forwarding thestream of log data to a first data plane and to a constraint plane;storing the stream of log data in the first data plane; extracting asubset of the stream of log data at the constraint plane in accordancewith a set of rules based on predefined queries of a real-time reportingservice; saving the subset of the stream of log data to a second dataplane; transmitting one or more metrics included in the subset of thestream of log data to the real-time reporting service; and providing theone or more metrics from the subset of the stream of log data to a userof the real-time reporting service.

Other aspects and advantages of the invention will become apparent fromthe following detailed description taken in conjunction with theaccompanying drawings which illustrate, by way of example, theprinciples of the described embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure will be readily understood by the following detaileddescription in conjunction with the accompanying drawings, wherein likereference numerals designate like structural elements.

FIG. 1 shows a diagram illustrating an exemplary server cluster suitablefor use with the embodiment described herein.

FIG. 2 shows a diagram illustrating an exemplary log intelligence intakeprocess.

FIG. 3 shows a diagram illustrating another exemplary log intelligencesystem.

FIG. 4 shows a more detailed diagram illustrating another exemplary logintelligence system similar in function to the one described in FIG. 3

FIGS. 5A-5C show depictions of exemplary log data and how it can bemodified as it is processed at the ingestion and constraint planes.

FIG. 6 shows a flow chart depicting a method for displaying metricsassociated with log data.

DETAILED DESCRIPTION

Certain details are set forth below to provide a sufficientunderstanding of various embodiments of the invention. However, it willbe clear to one skilled in the art that embodiments of the invention maybe practiced without one or more of these particular details. Moreover,the particular embodiments of the present invention described herein areprovided by way of example and should not be used to limit the scope ofthe invention to these particular embodiments. In other instances,hardware components, network architectures, and/or software operationshave not been shown in detail in order to avoid unnecessarily obscuringthe invention.

The efficiency and performance of a log intelligence system can benegatively impacted when the system it analyzes produces a large numberof logs. For example, when a user requests data from a primary datastore for logs spanning a long period of time, the log intelligencesystem may need to analyze a large amount of data to retrieve thedesired data to fulfill the user request. In a system having one or morequeries supporting the display of a real-time dashboard and thereporting of alerts, one way to reduce the amount of time needed to runthe one or more queries is to cache only the metrics needed to supportthe one or more queries.

One way to implement such a solution is to create an additional datastore associated directly with an analytics engine of the logintelligence system that contains only a subset of the log data that isbeing produced by an operational system. This subset of log data can bestored at the additional data store as part of the log data ingestionprocess. In this way, the subset of log data can be continuously updatedwith the most current log data generated by the operational system. Insome embodiments, the subset of data does not include the entirety ofpertinent log files but instead only includes specific metrics extractedfrom the pertinent log files. For at least this reason, an amount ofdata stored in the additional data store is substantially smaller thanthe amount of data stored in the primary data store. It should beappreciated that a size of the differential between the log data storedin the primary data store and the metric data stored in the additionaldata store can vary based on the scope and number of queries being runwithin the log intelligence system. Furthermore, having a dedicated dataplane for supplying the metrics needed to retrieve desired dashboardvisualizations and alerts also provides tangible benefits sincebackground processes such as log data backup do not negatively affectthe ability of the log intelligence system to provide rapid access tothe desired data defined by the one or more queries.

These and other embodiments are discussed below with reference to FIGS.1-6 ; however, those skilled in the art will readily appreciate that thedetailed description given herein with respect to these figures is forexplanatory purposes only and should not be construed as limiting.

FIG. 1 shows a block diagram illustrating an exemplary server cluster100 suitable for use with the embodiment described in this disclosure.Server cluster 100 can include hosts 102, 112, 122 and 132. While a fourhost system is shown for exemplary purposes it should be appreciatedthat server cluster 100 could include a larger or smaller number ofhosts. Each host 102-132 includes host hardware 110-140, which caninclude a designated amount of processing, memory, network and/orstorage resources. In some embodiments, each of the hosts provide thesame amount of resources, and in other embodiments, the hosts areconfigured to provide different amounts of resources to support one ormore virtual machines (VMs) running on the hosts. Each of the VMs can beconfigured to run a guest operating system that allows for multipleapplications or services to run within the VM.

Each of hosts 102, 112, 122 and 132 are capable of runningvirtualization software 108, 118, 128 and 138, respectively. Thevirtualization software can run within a virtual machine (VM) andincludes management tools for starting, stopping and managing variousvirtual machines running on the host. For example, host 102 can beconfigured to stop or suspend operations of virtual machines 104 or 106utilizing virtualization software 108. Virtualization software 108,commonly referred to as a hypervisor, can also be configured to startnew virtual machines or change the amount of processing or memoryresources from host hardware 110 that are assigned to one or more VMsrunning on host 102. Host hardware 110 includes one or more processors,memory, storage resources, I/O ports and the like that are configured tosupport operation of VMs running on host 102. In some embodiments, agreater amount of processing, memory or storage resources of hosthardware 110 is allocated to operation of VM 104 than to VM 106. Thismay be desirable when, e.g., VM 104 is running a larger number ofservices or running on a more resource intensive operating system thanVM 106. Clients 140 and 150 are positioned outside server cluster 100and can request access to services running on server cluster 100 vianetwork 160. Responding to the request for access and interacting withclients 140 and 150 can involve interaction with a single service or inother cases may involve multiple smaller services cooperativelyinteracting to provide information requested by clients 140 and/or 150.

Hosts 102, 112, 122 and 132, which make up server cluster 100, can alsoinclude or have access to a storage area network (SAN) that can beshared by multiple hosts. The SAN is configured to provide storageresources as known in the art. In some embodiments, the SAN can be usedto store log data generated during operation of server cluster 100.While description is made herein with respect to the operation of thehosts 110-140, it will be appreciated that those of hosts 110-140provide analogous functionality, respectively.

FIG. 2 shows a block diagram illustrating an exemplary log intelligenceintake process. Agent 200 can be incorporated into many different typesof environments (e.g., as a cloud infrastructure, an on premisesinfrastructure, or in a specific embodiment server cluster 100) totransmit log data that is generated in response to many different typesof events to data ingestion source gateway 202. For example, agent 200can generate logs stored in an events table that represent variousevents that are captured during normal or irregular operation of agent200. The logs could include any number of metadata and a time stamp thathelps to determine how often logs of particular types are generated. Themetadata could be used to help identify whether the logs are related todetected errors or more normal activity such as a login or file downloadevent. Data ingestion source gateway 202 is configured to forward logdata received from agent 200 to ingestion pipeline 204 and/or buffer206. Log data received at ingestion pipeline 204 is then forwarded on torouter 208, which distributes the log data to data plane 210. Log datacan be sent to buffer 206 when a rate at which log data is beingsupplied by agent 200 exceeds a rate ingestion pipeline 204 can handle.In such a situation, buffer 206 can take the form of a Kafka module ableto handle many extremely large streams of data. In some embodiments, theKafka module can be configured to distribute multiple streams of the logdata to separate computing resources to keep up with a rate at which thelog data is being produced. Such a situation may arise when the systemassociated with agent 200 is undergoing high usage and/or experiencinglarge numbers of errors or warnings. Data plane 210 can be organizedinto multiple shards that improve reliability of the data store but mayalso limit a rate at which the stored log data can be retrieved. In someembodiments, the data logs can also be backed up to a cloud service 212.Cloud service 212 can provide access to the log data during an onpremise server outage or be used to restore data lost due to equipmentfailure.

A user is able to retrieve relevant subsets of the log data from dataplane 210 by accessing user-facing gateway 214 by way of user interface216. Data representative of the log data is obtained by dashboardservice 218, alert service 220 and user-defined query module 222.Dashboard service 218 is generally configured to retrieve log data fromdata plane 210 within a particular temporal range or that has aparticular log type. Dashboard service 218 can include a number ofpredefined queries suitable for display on a dashboard display.Dashboard service 218 could include conventional queries that helpcharacterize metrics such as error occurrence, user logins, serverloading, etc. Alert service 220 can be configured to alter the user whenthe log data indicates a serious issue and user-defined query module 222allows a user to define custom queries particularly relevant tooperation of the application associated with agent 200.

With this type of configuration, dashboard service 218, alert service220 and user-defined query module 222 each route requests for data tosupport the alerts and queries to data plane 210 by way of router 208.Queries are typically run to retrieve the entire dataset relevant to thequery or alert in order to be sure time-delayed logs are not missed fromthe queries. In this way, the queries can be sure to obtain all datarelevant to the query.

FIG. 3 shows a block diagram illustrating another exemplary logintelligence system 300. In particular, agent 302 can be installedwithin operational system 304 and configured to transmit a stream of logdata generated by operational system 304 to ingestion pipeline 306. Insome embodiments, the connection between agent 300 and the ingestionpipeline 302 can be a direct connection or alternatively be transmittedacross a larger network. Ingestion pipeline 302 can be configured toperform basic formatting and parsing operations upon the log data priorto transmitting the log data to data plane 308. In some embodiments, thelog data stored in data plane 308 can be backed up to other serverslocated on premises or at a number of distributed cloud computingfacilities. Ingestion pipeline 306 can also be configured to providedata to analytics data storage 308. Analytics system 310 can include arobust set of filters that processes only the log data pertaining to acurrent set of metrics requested from real-time display system 310. Forexample, when processing the log data, any log data files failing tomatch one or more log data criteria can be discarded to save space andreduce access time to the log data stored on analytics data storage 308.In some embodiments, log data that is saved can be reduced in size byincluding only metrics currently being requested by real-time reportingservice 312. Saving only a subset of the log data relevant to what iscurrently being used by the real-time reporting service during the dataingestion process allows for much more rapid performance of the logintelligence system. In some embodiments, the speed of real-timereporting service 312 is increased by at least an order of magnitudewhen compared with a configuration similar to the one depicted in FIG. 2.

FIG. 4 shows a more detailed block diagram illustrating anotherexemplary log intelligence system 400 similar in function to the onedescribed in FIG. 3 . In particular, agent 402 is shown and as describedabove can be installed on an operational system. The operational systemgenerates a stream of log data that is transmitted by agent 402 toingestion plane 404. Ingestion plane 404 can optionally include a parser406 that helps to identify the various types and locations of metricsincluded in the stream of log data. In a specific embodiments, parsingthe log data can include the use of JSON parsing, followed by convertingun-structured data to structured data using Grok parsers and theapplication of machine learning to improve the performance of theparsing. Parser 406 is configured to transmit the stream of log data torouter 408 and to constraint plane 410. In some embodiments, parser 406is configured to transmit the stream of log data to router 408 and toconstraint plane 410 concurrently. Router 408 is responsible fordistributing the stream of log data to data plane 412. In the depictedconfiguration data plane 412 includes multiple shards. In such aconfiguration router 408 can be configured to help distribute the streamof log data to each of the shards of data plane 412. Data can beorganized in various manners within data plane 412.

In some embodiments, log data can be stored temporally so that logspulled within a particular time are distributed to a particular one ofthe shards. In some embodiments, the stream of log data is evenlydistributed across each of the shards to improve intake time.Unfortunately, by distributing the log data across multiple shards inthis manner, the performance of any queries is run against data plane412 is reduced since the queries would have to be run against each ofthe shards containing the desired log data. As described in FIG. 2 , logintelligence system 400 can include a buffer or Kafka module withiningestion plane 404 that is configured to process log data when the rateat which log data is provided overwhelms the system's ability to processit.

Ingestion plane 404 sends the same stream of log data to constraintplane 410 as it sends to router 408. In the depicted embodiment, ruleconfiguration service 414 of constraint plane 410 is used to establishrules 416, which are derived from the queries supporting variousreporting services. Rules 416 are then used to help identify which logsto extract from the stream of log data to display metrics of interest toan administrator or user of the operational system or to provide alertswhen particular events occur at the operational system. In someembodiments, the displayed metrics are defined by developer-definedqueries or queries built by users to fulfill a desired purpose. Thealerts can be pre-defined by the developer and/or setup by a user who isable to change or update the queries associated with the alert service.A user interface can be configured to allow a user to subscribe to oneor more alerts cued when the stream of log data indicates the occurrenceof a particular event or sequence of events meeting an establishedcriteria. The user interface also allows for the queries and/or alertsto be adjusted, added to or subtracted from.

Matching module 418 can be configured to process logs from a subset ofthe stream of log data that are determined to contain metrics matchingrules 416. For example, an algorithm incorporated within matching module418 and derived from rules 416 could be configured to perform asubstring search on the stream of log data to identify logs that includecertain keywords or terms such as error, login, critical and the like.In some embodiments, results from these substring searches could be usedby matching module 418 to create one or more metrics, e.g., the numberof errors or logins that have occurred over a particular period of time.The algorithm could also be configured to harvest text adjacent to orproximate to the searched for terms for the generation of other types ofmetrics. It should be noted that the matching module 418 can also employnon-string searches geared toward searching for particular attributesidentified during the ingestion process.

Matching module 418 is then responsible for forwarding at least aportion of the subset of the stream of log data extracted from the logdata to metric system 420. In some embodiments, this subset includesonly logs with relevant metric data and in other embodiments, only themetric data is sent on to metric system 420. Metric system 420 caninclude its own data storage plane and take the form of a log analyticsprogram such as Wavefront by VMWare®, which is able to catalog andorganize the subset of the stream of log data. The log analytics programis configured to transmit the organized data to real-time reportingservices such as, e.g., dashboard service 422, alert service 424 anduser-defined query service 426. In some embodiments, communicationbetween services 422, 424, 426 and metric system 420 are two-waycommunications that allow the different reporting services to requestupdated results at particular intervals and/or in particular formats.User 428 is able to request adjustments to dashboard service 422, alertservice 424 and user-defined query service 426. As adjustments definedby users, these adjustments normally are expressed initially in the formof a log-based rule. For example, log-based rules are typicallyexpressed in human-readable plain text whereas metric-based rules areconfigured to be read quickly and efficiently by computers and oftencontain additional metadata such as specific field names that might bedefined during the parsing process. Services 422-426 can be configuredto transmit log-based rules setup by user 428 to transformer 430.Transformer 430 can be configured to transform the log-based rules intometric-based rules formatted to be understood by metric system 420. Insome embodiments, transformer 430 can also be configured to updatemetric-based rules 416 being implemented by matching module 418 based onlog-based rules provided by services 422-426. The conversion oflog-based rules to metric-based rules allows services 422-426 toefficiently communicate requests for new or updated information directlyfrom metric system 420.

When the adjustment requests require historical data not alreadycontained within metric system 420, a request can be sent to data plane412 requesting the historical data be transferred to metric system 420.The transferred log data can pass through constraint plane 410 toextract portions of the requested log data entries that are not neededfor the new query or alert. Since rules 416 are updated as part of thisprocess, further data needed to update the new queries or alerts can beprovided as part of the previously described data ingestion process sothat no further data needs to be requested from data plane 412.Constraint plane 410 and metric system 420 can be collectively referredto as the analytics engine of log intelligence system 400 as thesemodules are responsible for identifying the relevant log data andproviding structured data to the dashboard and alert services.

FIGS. 5A-5C show depictions of exemplary log data and how it can bemodified as it is processed at the ingestion and constraint planes. Inparticular, FIG. 5A shows a depiction of exemplary log data 500 made upof multiple unstructured logs 501-503 prior to its arrival at ingestionplane 404. Other than the data being organized into different logsrepresented by logs 501-503 the data can be largely without structure.FIG. 5B shows how after log data 500 is processed by parser 406 numerousmetrics including event time can be identified. Other examples ofmetrics contained within each of the logs includes event type. There canbe numerous event types including, e.g., transaction logs, applicationlogs, system logs, event logs and the like. An event type metric can beused to broadly separate out and filter one type of log from another.For example, a particular type of dashboard service might only beinterested in tracking the number of application logs identifyingcritical errors over a particular period of time. Given such asituation, FIG. 5C shows how Log 502 can be discarded from the stream oflog data at constraint plane 410 as either not being an application logor not being an application log directed to a critical error. One ormore metrics can also be removed from each of the remaining logs byconstraint plane 410 as depicted in FIG. 5C. In other embodiments,instead of saving logs from which unneeded metrics have been removed,the relevant metrics can instead be extracted from the logs. It shouldbe appreciated that metric system 420 could also be configured to reducea subset of logs that included the relevant metrics to just the relevantmetrics themselves in order to streamline the amount of storage taken bythe data and to improve metric access speeds.

For the critical error query described above, metrics such as user name,application name and other specific details about the error would not beneeded to drive a dashboard visualization that only relates to thenumber of critical errors detected over a particular period of time. Fora system in which the metrics were extracted by constraint plane 410 atleast the event time, event type and event identifier would be extractedfor each log referencing a critical error. It should be noted older logdata can be retained by metric system 420 to allow a duration of queriesto be adjusted rapidly if a user of the dashboard service decides toreview different older periods of time. In some embodiments, the sizereduction achieved by saving only metric data results in performanceincreases of greater than an order of magnitude compared with theconfigurations described in FIG. 2 .

FIG. 6 shows a flow chart 600 depicting a method for displaying metricsassociated with log data. At 602, a stream of log data is received froman agent associated with an operational system. The agent can beinstalled or in communication with the operational system and configuredto transmit the stream of log data generated by the operational systemto the ingestion plane of a log intelligence product. At 604, theingestion plane forwards the stream of log data to a first data planeand to a constraint plane. In some embodiments, the ingestion plane caninclude a parsing module that identifies metrics stored within thestream of log data prior to forwarding the stream of log data. At 606,the stream of log data is saved within the first data plane. The firstdata plane can be formed from a number of shards across which the streamof log data can be distributed. At 608, a subset of the stream of logdata is extracted to reduce a size of the stream of log data to a moremanageable size. On average, operational systems can generate millionsof logs in a 24 hour period, making a subset more manageable and quickerto access. At 610, the subset of the stream of log data is saved to asecond data plane. The data plane can be a component of a log analyticsprogram configured to organize and distribute the subset of the streamof log data to one or more real-time reporting services. At 612, aportion of the subset of the stream of log data is transmitted to adashboard service. Other portions of the subset of the stream of logdata can be transmitted to other services such as an alert service or auser-defined query service. At 614, the portion of the subset of thestream of data is used to display one or more metrics to a user of thelog intelligence product. Because the stream of data stored at thesecond data plane is much smaller than the stream of data located at thefirst data plane and not distributed across multiple shards, an amountof time needed to access the data can be substantially smaller than thetime needed to query the stream of data stored in the first data plane.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the describedembodiments. However, it will be apparent to one skilled in the art thatthe specific details are not required in order to practice the describedembodiments. Thus, the foregoing descriptions of specific embodimentsare presented for purposes of illustration and description. They are notintended to be exhaustive or to limit the described embodiments to theprecise forms disclosed. It will be apparent to one of ordinary skill inthe art that many modifications and variations are possible in view ofthe above teachings.

What is claimed is:
 1. A computer implemented method for displayingmetrics associated with log data, the computer implemented methodcomprising: at a data plane: receiving a stream of log data generated byan operational system; extracting a subset of log data from the receivedstream of log data, the extracting performed in accordance with a set ofrules based on predefined queries of a real-time reporting service;storing the extracted subset of log data; and transmitting one or moremetrics included in the extracted subset of log data to a real-timereporting service.
 2. The computer implemented method of claim 1,further comprising: providing the one or more metrics to a user of thereal-time reporting service.
 3. The computer implemented method of claim1, wherein the data plane is a first data plane, and the receivingfurther comprises receiving the stream of log data from a second dataplane different from the first data plane.
 4. The computer implementedmethod of claim 3, wherein the first data plane is a constraint plane,and the second data plane is an ingestion plane configured to receivethe stream of log data from the operational system and to forward thestream of log data to the first data plane.
 5. The computer implementedmethod of claim 1, wherein the storing further comprises, of thereceived stream of log data, storing only the extracted subset of logdata and discarding a remainder of the log data besides the extractedsubset of log data.
 6. The computer implemented method of claim 1,wherein the subset of log data includes only logs of the received logdata that include metrics specified by the set of rules.
 7. The computerimplemented method of claim 1, wherein the storing further comprisesstoring the entire extracted subset of log data at a single shard. 8.The computer implemented method of claim 1, wherein the real-timereporting service comprises a dashboard service.
 9. The computerimplemented method of claim 8, further comprising initiating a graphicaldisplay of the one or more metrics, the graphical display illustrating anumber of occurrences of an event type over a predefined period of time.10. The computer implemented method of claim 1, wherein the subset oflog data includes only metric data.
 11. The computer implemented methodof claim 1, further comprising updating the set of rules in response toan update to one or more of the queries of the real-time reportingservice.
 12. The computer implemented method of claim 11, wherein thedata plane is a first data plane, the method further comprisingrequesting historical data from a second data plane different from thefirst data plane when metrics specified by the one or more queries arenot stored at the first data plane.
 13. The computer implemented methodof claim 12, further comprising, at the first data plane, receiving thehistorical data from the second data plane.
 14. The computer implementedmethod of claim 13, further comprising extracting, from the historicaldata, one or more metrics specified by the queries.
 15. A non-transitorycomputer-readable storage medium storing instructions configured to beexecuted by one or more processors of a computing device cause thecomputing device to carry out steps that include: at a data plane:receiving a stream of log data generated by an operational system;extracting a subset of log data from the received stream of log data,the extracting performed in accordance with a set of rules based onpredefined queries of a real-time reporting service; storing theextracted subset of log data; and transmitting one or more metricsincluded in the extracted subset of log data to a real-time reportingservice.
 16. The non-transitory computer-readable storage medium ofclaim 15, wherein the data plane is a first data plane, and thereceiving further comprises receiving the stream of log data from asecond data plane different from the first data plane.
 17. Thenon-transitory computer-readable storage medium of claim 16, wherein thefirst data plane is a constraint plane, and the second data plane is aningestion plane configured to receive the stream of log data from theoperational system and to forward the stream of log data to the firstdata plane.
 18. A computer system, comprising: one or more processors;and memory storing one or more programs configured to be executed by theone or more processors, the one or more programs including instructionsfor: at a data plane: receiving a stream of log data generated by anoperational system; extracting a subset of log data from the receivedstream of log data, the extracting performed in accordance with a set ofrules based on predefined queries of a real-time reporting service;storing the extracted subset of log data; and transmitting one or moremetrics included in the extracted subset of log data to a real-timereporting service.
 19. The computer system of claim 18, wherein the dataplane is a first data plane, and the receiving further comprisesreceiving the stream of log data from a second data plane different fromthe first data plane.
 20. The computer system of claim 19, wherein thefirst data plane is a constraint plane, and the second data plane is aningestion plane configured to receive the stream of log data from theoperational system and to forward the stream of log data to the firstdata plane.